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ABSTRACT 

The effectiveness of four technical 
assistance-oriented services/products of one regional educational 
laboratory was studied. The services/products studied included a 
thinking skills activities workbook for classroom teachers, a 
videotape for rural educators on exemplary software for developing 
the thinking skills of at-risk students, a research report for urban 
school administrators on the interrelationships between student and 
teacher commitment, and a Chapter 1 program improvement and 
monitoring process (monitoring instrument, training, and ongoing 
consultation in the monitoring and improvement of Chapter 1 
programs). A tracer study tracked the dissemination and use of 
services/products through exchanges of information with the external 
evaluators, following which the service/product providers and 
primary, secondary, and tertiary recipients were interviewed. 
Recipients were contacted by telephone by the evaluators between 6 
months and 1 year after receipt of the service/product. The workbook 
and the program improvement and monitoring process were found to be 
more successful than the other two services/products in their degree 
of dissemination, assessment, and use. The workbook and the process 
met a defined need, they had a clearly specified target audience, 
their purpose and potential use were focused and unambiguous, ana 
they could be used without extensive further direction,* the videotape 
and the research report did not meet these criteria. Two re/erences 
are included. (RLC) 
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Use of R&D Laboratory Services and Products for School Improvement 

Joan Le Buttram, Research for Better Schools 
Sheila Rosenblum and Nancy Brigham f Rosenblum Brigham Associates 



Introduction 

The impact of regional educational laboratories in fostering and 
strengthening state and local efforts to improve schooling for all students, 
especially those at risk, has come under increasing scrutiny over the past 
decade. With the growing numbers of at-risk students and fairly level funding 
for the ten laboratories, their ability to reach out and provide critical 
services that will directly impact on students has been stretched* As a 
result, the laboratories have chosen to devote some of their limited resources 
to develop services and products that can be used by others to improve 
schooling more directly* The dissemination and effectiveness of these 
technical assistance-oriented services and products is not widely known. This 
study thus attempted to collect information on a sample of one laboratory's 
services and products as they were disseminated and used* This information, 
in turn, will be used to strategically plan future regional services and 
products* 

The general approach to this effort was based on a framework for 
studying knowledge utilization and the methodology of tracer studies (Louis, 
Kell, Dentler, Corwin, & Herriot, 1984) . In very simplistic terms, knowledge 
utilization occurs when aa exchange of purposefully prepared information is 
communicated to a set of recipients (Louis, et ai, 1984). in this particular 
case, knowledge utilisation occurs when laboratory clients (e.g*, state 
education staff, intermediate agency staff, central office and building 
administrators, classroom teachers) use laboratory services and products with 
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other educators who work more directly to Improve schooling. These events 
help to compound the laboratory's efforts to improve schooling in the region. 

Study Methodology 

In this particular case, the tracer study methodology will track the 
dissemination and use of services and products (or exchanges of information} 
once they leave the laboratory (Louis, et al, 1984). The basic features of 
this approach involve the selection of services and products, interviews with 
the providers and a sample of primary, secondary, and tertiary recipients of 
the service or product with whom the information is shared* This methodology 
thus helps track the flow of information through the network of original 
recipients of services or products to others in the educational system. 
Tracer Study Exchanges of Information 

The particular laboratory services and products included in this set of 
tracer studies were selected in the late spring of 1991* Although over 15 
different services or products were initially discussed at a laboratory staff 
meeting, eventually the sample was narrowed to four and included the 
following: 

• a thinking skills activities workbook for classroom teachers 

« a video tape for rural educators on exemplary software for developing 
the thinking skills of at-risk students 

0 a research report for urban school admir Istrators on the inter- 
relationships between student ard teacher commitment 

# a Chapter 1 program improvement and monitoring process, which 
includes a monitoring instrument, training, and ongoing consultation 
in the monitoring and improvement of Chapter 1 programs* 

These services and products were selected to provide a mix of dissemination 

strategies, contents, and clients. They also featured particular approaches 

that the laboratory was considering reusing in future work in the region. 



Tracer Study Samples 

A list of first level recipients (i.e., those recipients who had 
received the service/product directly from RBS) was generated for each service 
or product included in the tracer studies. A sample of first level recipients 
was randomly selected for each service or product* The size of the four first 
level samples varied depending on the size of the original pool of recipients. 
The first two products involved dissemination to hundreds of clients and so 
approximately 12-14 percent of the first level sample was contacted. Because 
of the limited dissemination of the research report (i.e., approximately 50 
recipients) , the tracer study sample was enlarged to include about 34 percent 
of the recipients. The Chapter 1 program improvement and monitoring process 
sample comprised 10 percent of the state monitors and 15 percent of the 
districts completing monitoring* First level recipients were asked to 
identify others (i.e., second level recipients) with whom they had shared 
laboratory services or products. Similarly, second and then third lev*»l 
recipients also were asked to identify others* Given the fairly small 
numbers, (i.e., less than 15), all second, third, and fourth level recipients 
were included in the tracer study samples. 
Tracer Study Procedures 

Once the selection of laboratory services and products was completed, 
the initial laboratory staff providers were interviewed by one of the third 
party, external evaluators {i.e., the second and third authors of this paper). 
These interviews focused on determining the history and goals of the service 
or product, the reason for its initiation, a full description of the planning 
and implementation process, and their assessment of the quality and usefulness 
of the service or product. In essence, laboratory staff provided contextual 
information for the subsequent interviews of recipients. 
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Recipients were contacted by telephone by the external evaluators. 
Interviews typically were conducted between 6 months and one year after they 
received the service or product. This allowed sufficient time for the service 
or product to be disseminated further and for the service or product to be 
used in a meaningful way. During these interviews, the external evaluators 
asked the recipients questions on: 

• their work setting 

• their recollection of having received the service or product 
0 the content of the service or product 

• their assessment of the quality of the service or product 

• a description and assessment of the assistance received 

• ways in which the service or product was used 

• ways in which the recipients* organization may have benefited from 
the service or product 

e ways in which the service or product was processed, discussed, and 
shared in their organizations 

• the degree to which recipients typically look for new ideas and their 
usual sources for seeking such information. 

They also were asked to identify other people with whom they shared the 

laboratory materials (i.e., next level recipients). 

Information obtained from these interviews was analyzed to answer a 

series of critical questions. Foremost was the determination of the 

effectiveness of the individual service or product in reaching and being used 

by regional educators to help improve schooling- In addition, analyses were 

conducted to compare the results for each service or product so that critical 

features in the services or products themselves, or their dissemination 

strategies, could be identified. 
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Tracer Study Results 
As might be expected, the effectiveness of the four services or products 
varied considerably ♦ The results for each are presented separately. 
Following these individual presentations, several overall comparisons are 
made. 

Thinking Skills Activity Workbook / 

The workbook was very successful, based partially on its ability to 

reach the proper audience. It relied on a reactive dissemination strategy, 

individuals purchased the workbook through the laboratory's publication 

catalog. The workbook scored high in its use and effectiveness. Looking at 

appropriate use (which includes both direct use in the classroom and placement 

in a professional library as a resource material), the workbook was used 

appropriately by 89 percent of the first and second level recipients* It was 

used directly in the classroom by 68 percent of the first and second level 

recipients. All but two of them rated it as very effective. Respondents 

tended to express their views about the workbook enthusiastically as 

illustrated in the two (of many) statements quoted below. 

This is one staff development specialist who will recommend 
the book highly and put it on a list of printed and visual 
resources for use in the schools. Teachers are not 
interested in theories; they want something they can use. 
Compared to other material, this is not didactic, but 
rather practical and fun. 

I . the principal here. I ordered the book at the request 
of the Chapter 1 teacher* I looked it over and loved it. 
When the teacher left the school, she wanted to take it 
with her but I said "oh no, we paid for it and we want to 
keep it. " 

The thinking skills activity workbook clearly reached its target audience, was 
passed on, and used by practitioners in their classrooms. 
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Thinking Skills Video Tape 

The video tape was disseminated using a proactive strategy. Rural 
superintendents and others were sent a copy of the video tape along with a 
letter explaining its purpose* The tracer study results showed that it was 
used directly by 45 percent of the combined first and second level recipients, 
receiving greater use at the second level than at the first. In addition, 
82 percent of the first and second level recipients handed the video tape on 
to someone else, indicating that recipients did try to make sure that the 
video tape was received by the right audience. 

In spite of this valiant dissemination effort, the effectiveness of the 
video tape must be considered mixed. Some recipients thought the video tape 
was : 

.•.on target for our school although there's nothing major 
that's new. It dealt with a specific population that we 
are familiar with, people have always thought that thinking 
skills are only for smarter kids, but now we use the idea 
across the curriculum. Might be good for parents because 
ideas might be new to them. 

However, other felt that the video tape content did not reflect the rural 

school emphasis suggested in the title. Still others questioned the quality 

of the video tape and reported that they! 

...had trouble staying awake* I guess it's as good as 
anything else that introduces a canned program that can be 
used in school districts if they have computers and want to 
purchase materials. 

Overall, there were significant questions from many recipients about the video 
tape's emphasis on rural school settings and its quality. 
Research Report on High School Commitment 

Both reactive and proactive strategies wera used to disseminate the 
research report. In terms of the former, the report was sent to educators 
purchasing a copy through the laboratory's publication catalog. The report 



also was sent to the five urban superintendents who had initially requested 
the study. The results for each dissemination strategy are presented 
separately. 

The research report was not used by any of the first level recipients 

who had purchased the report. Twelve of the 13 turned out to be "dead ends* 

in that they did not remember the report at all, or if they did remember, they 

had not read it, One of the recipients who recalled receivi^ the report but 

had not yet read it, made the following comment: 

I have always been happy with RBS materials, but haver, 't 
gotten around to this one yet. From the outward appe 
ance, this book looks like a dissertation* This may ba 
part of the reason I have not read it yet. I find thet the 
easier a book or manual is to read (in appearance and 
content), the more likely it will actually, get read, I 
prefer material that has "at least one immediate applica- 
tion spelled out. * If it is up to the reader to go to the 
application level, then it is less likely to be of value. 

The one person included in the tracer study sample who had read the book, a 
teacher in a "school without walls * had very mixed reactions to it. He found 
the summary and recommendations at the beginning to be interesting, but the 
remainder to be too statistical and dry. He also found it to be too "problem- 
oriented" and would have preferred to know more about what is working. 

Tracing the use of the research report via the five superintendents 
turned out to be very difficult. In two of the five urban districts, there 
had been extensive turnover at the senior level and so there was no point of 
entry for this study. Another superintendent did not respond to numerous 
requests for an interview. In the other two districts, the superintendents 
praised the report, but could not pinpoint any use of the study. 

Together, these findings suggest that the presentation of the findings 
and dissemination strategy were not very successful. Educational 



administrators were not able to use the study's findings in their current form 
to help lead improvement efforts in their schools. 
Chapter 1 Program Improvement and Monitoring Process 

The methodology used for the Chapter 1 program improvement and moni- 
toring process was somewhat complex in that it included both staff members who 
conducted Chapter 1 monitoring and school district staff members whose Chapter 
1 programs were monitored. Unlike the earlier studies that traced the 
dissemination of a discrete product from one level of recipients to the next, 
this tracer study interviewed two groups of recipients who used laboratory 
products or services in carrying out their Chapter 1 program monitoring 
responsibilities. The results for each are discussed below. 

Monitors of Chapter 1 programs . As noted above, 25 individuals who 
served as monitors of Chapter 1 programs were interviewed; half of the group 
was recommended by personnel from the state department of education, the other 
half were selected randomly from the roster of current Chapter l monitors. 
They were interviewed about the training they received, the monitoring 
approach, and the quality of the monitoring instrument and process. Although 
many offered substantive feedback to improve the monitoring process, they 
generally endorsed the importance and value of the monitoring process devised 
collaboratively by the laboratory and the state department of education. They 
perceived the emphasis on placing Chapter 1 within the entire educational 
context as essential to the program's success, and they tried hard to fulfill 
their role as purveyors of this practice. 

School di stricts monitored using this process . Telephone interviews 
were conducted with 25 educators in eight school districts located throughout 
the state. The interviewees were chosen from suggestions made by monitors who 
visited the districts. In each, the Chapter 1 coordinator was interviewed, 
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along with others including Chapter 1 reading teachers, principals, assistant 
superintendents, and superintendents* They were interviewed about their 
preparations for the monitoring visit, the monitoring approach, the quality of 
the instrument and monitoring process, and the outcomes of the monitoring 
visit. 

For most of the districts, the recent monitoring process was positive. 
Staff liked the holistic approach developed by the laboratory and the state 
department of education. They approved of the monitoring instrument, and 
especially enjoyed the opportunity to prepare for the monitoring visit with 
self-response forms. Although they reported that the monitoring process 
helped school personnel to clarify and refocus their Chapter 1 objectives (a 
primary purpose of the revised monitoring process), none had made any 
significant changes to their program in response to the monitoring visit. 

Tracer Study Suiranary and Conclusions 

The first three products (i.e., the thinking skills activity workbook, 
the thinking skills video tape, and the research report on high school 
commitment) differed in their extent of dissemination and use as well as theii 
users* estimates of their effectiveness. In order for these products to be 
effectively used, they must first reach the hands of their intended audiences. 
Reactive and proactive dissemination strategies were used to distribute the 
three; the workbook was disseminated by responding to orders through a 
laboratory's publication catalog or other information media (reactively) , the 
video tape was distributed proactively through mailings to superintendents and 
others initiated by RBS, and the research report was disseminated using both / 
reactive {publications catalog) and proactive (sent to urban superintendents) 
strategies* None of these methods was foolproof — people can order a book 
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and forget they have it. receive a video and misplace it, and ignore a 
research report that arrives on their desks. 

Nevertheless, it appears that the thinking skills workbook navigated the 
treacherous channels of dissemination more successfully than the video tape or 
research report in terms of reaching its audience and being judged effective. 
The reasons for its successful dissemination appear to be: 

• the publications catalog description represented the product 
accurately and non- ambiguously 

• the target audience was abundantly clear, even from the title 

• its use was clear and apparent without additional written 
instructions . 

This is in contrast to the video tape in which the target audience was not 
clearly defined, the title was somewhat misleading, and the use was not clear 
despite the suggestions for use that went out with it to superintendents. It 
is important to emphasize the difference between a target audience (those who 
use a product directly) and target recipients (such as the superintendents), 
who are really intermediates in that they are expected to figure out the 
proper audience and pass the product on to them. While it was appropriate to 
send the video tape to superintendents as the major contact point in rural 
areas, it would have been useful to suggest in the accompanying letter who 
should use the video tape as well as how to use it. 

The fate of the research report also can be attributed to the problems 
of ambiguous description, targeting, and lack of clarity or specificity as to 
how it could be used. In the publication catalog, the report was represented 
as a guide for improvement, targeted to administrators. In actuality, it is a 
summary of research, with recommendations based on findings. It would take a 
sophisticated "knowledge user 9 to be able to use the recommendations in an 
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active program of school improvement. The path to lead recipients in its use 
is not well-marked. 

It is difficult to draw direct comparisons of the above three products 
with the fourth, the Chapter 1 prograr improvement and monitoring process. In 
one very real sense, this service had a captive audience state Chapter 1 
monitors and programs wt^rf were required to use the revised Chapter 1 program 
improvement and monitoring process* Nevertheless, the results of tracer 
study interviews suggest that the revised process was being accepted and used 
extensively throughout the state. Its success was certainly due, in part, to 
its clearly defined purpose and role in the state f s Chapter 1 monitoring 
process . 

In conclusion, the results of the four tracer studies showed that two of 
the laboratory services and products were more successful than the other two 
in their degree of dissemination, assessment, and use. These are the thinking 
skills activity workbook and the Chapter 1 program improvement and monitoring 
process. Although different from each other in scope, substance, and 
complexity, the two shared certain basic factors: 

« they met a real and clearly defined need 

• they had a clearly specified target audience 

• their purpose and potential use was focused and unambiguous 

• they could be used without extensive further direction. 

The two other products, the thinking skill video tape and the research report 
on high school commitment , were not as successful and did not meet these 
criteria as well. 

The tracer study results have been shared with all of the laboratory 
staff. To date, the findings from the first year tracer studies have affected 
work by the laboratory or its clients in two ways. First, the two RBS staff 
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members who developed the original video tape have revised their plans for 
dissemination of a second video tape, incorporating ideas gleaned from the 
above tracer studies. Secoud, the state Chapter 1 coordinator has used 
feedback obtained from both Chapter 1 monitors and districts being monitored 
to modify an upcoming state Chapter 1 monitoring cycle. Both of these uses 
occurred within weeks of presenting the first year tracer studies results. It 
will be important to track the more long-term impact of the results on the 
development and dissemination of future laboratory products and services. 
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